Training method for image processing model, image processing method, network device, and storage medium

ABSTRACT

Embodiments of this application disclose a training method using image processing model for processing blurry images. The method includes obtaining a sample pair comprising a clear image and a corresponding blurry image; the sharpness of the clear image being greater than a preset threshold, the sharpness of the blurry image being less than the preset threshold; activating the image processing model to perform sharpness restoration on the blurry image to obtain a restored image; and updating network parameters of a first network and network parameters of a second network in the image processing model according to the restored image and the clear image to obtain a trained image processing model; the network parameters of the first network and the network parameters of the second network meeting a selective sharing condition indicating whether the network parameters between the first network and the second network are shared or independent.

RELATED APPLICATIONS

This application is a continuation application of PCT Application No.PCT/CN2020/077699, entitled “METHOD FOR TRAINING IMAGE PROCESSING MODEL,IMAGE PROCESSING METHOD, NETWORK DEVICE, AND STORAGE MEDIUM” and filedon Mar. 4, 2020, which claims priority to Chinese Patent Application No.201910259016.7, entitled “TRAINING METHOD FOR IMAGE PROCESSING MODEL,IMAGE PROCESSING METHOD, AND RELATED DEVICE”, and filed with theNational Intellectual Property Administration, PRC on Apr. 1, 2019. Thetwo applications are both incorporated herein by reference in theirentirety.

FIELD OF THE TECHNOLOGY

This application relates to the field of Internet technologies,specifically, to the field of image processing technologies, and inparticular, to a training method for an image processing model forprocessing blurry images, an image processing method, a trainingapparatus for an image processing model for processing blurry images, animage processing apparatus, a network device, and a computer storagemedium.

BACKGROUND OF THE DISCLOSURE

Image deblurring is an important research direction of image processing,and aims to restore detail information lost in blurry images. With theresearch advancement of neural network models, image deblurring methodsbased on an image processing model have achieved better effects thanconventional methods. The image processing model is a neural networkmodel used for performing image deblurring on the blurry images toobtain clear images. How to obtain an image processing model withperfect performance through model training is particularly important forthe effect of subsequent image deblurring. In existing model trainingmethods, it is generally considered that a blurry image is composed of aplurality of different blurry regions, and convolution model assumptionsare applied on the different blurry regions to restore the differentblurry regions to clear images in the different regions respectively, tofurther train the image processing model. Specifically, it is necessaryto segment the blurry image into different regions first, thencontinuously perform two operations of iterative convolution kernelestimation and image deconvolution on the different regions to graduallyoptimize a deblurring effect of each region, and finally synthesize theregions after deblurring to obtain a complete clear image.

SUMMARY

An embodiment of this application provides a training method for animage processing model for processing blurry images, performed by anetwork device, the image processing model comprising a first networkand a second network; the first network and the second network beingcodec networks with different scales; the sizes of the scalescorresponding to the measurements of the sharpness of to-be-processedblurry images. The method includes obtaining a sample pair for training,the sample pair comprising a clear image and a blurry imagecorresponding to the clear image; and the sharpness of the clear imagebeing greater than a preset threshold, and the sharpness of the blurryimage being less than the preset threshold; activating the imageprocessing model to perform sharpness restoration on the blurry image toobtain a restored image; and updating network parameters of the firstnetwork and network parameters of the second network in the imageprocessing model according to the restored image and the clear image toobtain a trained image processing model. The network parameters of thefirst network and the network parameters of the second network meet aselective sharing condition, and the selective sharing conditionindicate the network parameters between the first network and the secondnetwork are shared or independent.

An embodiment of this application further provides an image processingmethod, performed by a network device. The method includes obtaining ato-be-processed original image, the sharpness of the original imagebeing less than a preset threshold; activating an image processing modelfor processing blurry images to perform sharpness restoration on theoriginal image to obtain a target image, the sharpness of the targetimage being greater than the preset threshold, the image processingmodel at least comprising a first network and a second network; thefirst network and the second network being codec networks with differentscales; the sizes of the scales corresponding to the measurements of thesharpness of to-be-processed blurry images; and the network parametersof the first network and the network parameters of the second networkmeeting a selective sharing condition, and the selective sharingcondition indicating the network parameters between the first networkand the second network are shared or independent; and outputting thetarget image.

An embodiment of this application provides a training apparatus for animage processing model for processing blurry images, the imageprocessing model comprising a first network and a second network; thefirst network and the second network being codec networks with differentscales; the sizes of the scales corresponding to the measurements of thesharpness of to-be-processed blurry images. The apparatus includescomprising a processor, and a memory connected to the processor, thememory storing machine-readable instructions, and the machine-readableinstructions being executable by the processor to: obtain a sample pairfor training, the sample pair comprising a clear image and a blurryimage corresponding to the clear image; and the sharpness of the clearimage being greater than a preset threshold, and the sharpness of theblurry image being less than the preset threshold; activate the imageprocessing model to perform sharpness restoration on the blurry image toobtain a restored image; and update network parameters of the firstnetwork and network parameters of the second network in the imageprocessing model according to the restored image and the clear image toobtain a trained image processing model. The network parameters of thefirst network and the network parameters of the second network meet aselective sharing condition, and the selective sharing conditionindicating the network parameters between the first network and thesecond network, are shared or independent.

An embodiment of this application further provides a non-transitorycomputer-readable storage medium, storing a plurality of instructions,the instructions being configured to be loaded by a processor, toperform the training method for an image processing model and the imageprocessing method according to the embodiments of this application.

BRIEF DESCRIPTION OF THE DRAWINGS

To describe the technical solutions in the embodiments of thisapplication more clearly, the following briefly introduces theaccompanying drawings required for describing the embodiments.Apparently, the accompanying drawings in the following description showonly some embodiments of this application, and a person of ordinaryskill in the art may still derive other drawings from these accompanyingdrawings without creative efforts.

FIG. 1 is a schematic structural diagram of an image processing modelaccording to an embodiment of this application.

FIG. 2A is a schematic structural diagram of a first-order residualfunction according to an embodiment of this application.

FIG. 2B is a schematic diagram of a comparison between a second-ordernested skip connection structure corresponding to a second-orderresidual function and a series connection structure according to anembodiment of this application.

FIG. 2C is a schematic structural diagram of a third-order nested skipconnection structure corresponding to a third-order residual functionaccording to an embodiment of this application.

FIG. 2D is a schematic diagram of an internal structure of a featuretransformation unit according to an embodiment of this application.

FIG. 3A is a schematic diagram of a scenario of a training method for animage processing model according to an embodiment of this application.

FIG. 3B is a schematic flowchart of a training method for an imageprocessing model according to an embodiment of this application.

FIG. 3C is a flowchart of sharpness restoration according to anembodiment of this application.

FIG. 4 is a schematic diagram of a research result of blurry imagesaccording to an embodiment of this application.

FIG. 5A is a schematic diagram of allocation of network parametersaccording to an embodiment of this application.

FIG. 5B is a schematic diagram of another allocation of networkparameters according to an embodiment of this application.

FIG. 6A is a schematic flowchart of a training method for an imageprocessing model according to another embodiment of this application.

FIG. 6B is a flowchart of a method for obtaining a sample pair fortraining according to an embodiment of this application.

FIG. 6C is a flowchart of a method for activating an image processingmodel to perform sharpness restoration on a blurry image in step S602according to an embodiment of this application.

FIG. 7 is a schematic flowchart of an image processing method accordingto an embodiment of this application.

FIG. 8 is a schematic structural diagram of a training apparatus for animage processing model according to an embodiment of this application.

FIG. 9 is a schematic structural diagram of an image processingapparatus according to an embodiment of this application.

FIG. 10 is a schematic structural diagram of a network device accordingto an embodiment of this application.

DESCRIPTION OF EMBODIMENTS

The following clearly and completely describes technical solutions inembodiments of this application with reference to the accompanyingdrawings in the embodiments of this application.

The AI technology is a comprehensive discipline, and relates to a widerange of fields including both hardware-level technologies andsoftware-level technologies. The basic AI technologies generally includetechnologies such as a sensor, a dedicated AI chip, cloud computing,distributed storage, a big data processing technology, anoperating/interaction system, and electromechanical integration. AIsoftware technologies mainly include several major directions such as acomputer vision technology, an audio processing technology, a naturallanguage processing technology, and machine learning/deep learning.

Currently, deep learning is a technology of machine learning and one ofresearch fields. Artificial intelligence (AI) is implemented in acomputer system by building an artificial neural network with ahierarchical structure.

Due to successful application of deep learning (DL) in the field ofvision, researchers also introduce the DL to the field of imageprocessing. A deep learning neural network model is trained by using alarge quantity of training images to enable the model to perform imageprocessing, for example, process blurry images.

Image blurring is a common problem in image capture. In an example, whena user is in a dynamic scene or a relatively dark environment, amovement of an object in the dynamic scene and/or a movement of arecording camera may cause images obtained by recording to be blurry tovarious degrees. In another example, when a user records a targetobject, slight shaking of a hand of the user may also cause imagesobtained by recording to be blurry to various degrees. When facing ablurry image obtained by recording, the user usually chooses tore-record to obtain a clear image. The blurry image herein refers to animage of which the sharpness is less than a preset threshold, and theclear image refers to an image of which the sharpness is greater thanthe preset threshold. The sharpness refers to a degree of clearness ofdetail textures and borders thereof in the image. However, due tovarious factors such as camera movement, object movement, and handshaking, the user still cannot obtain a clear image after a plurality oftimes of re-recording. In addition, in some recording scenes ofinstantaneous snapshots, a user usually does not have a second chance ofrecording. For example, in a scene of recording a landscape outside awindow on a high-speed moving car/train, or in a scene of recording afast-moving object in a static scene, the user does not have a chance ofre-recording.

In the process of processing a blurry image, because an actual recordingscene of the blurry image is extremely complex, including a plurality offactors such as camera movement and object movement in a recordingscene, existing model training methods cannot satisfy convolution modelassumptions on all blurry regions in movement, resulting in poor imagedeblurring performance of the image processing model obtained throughtraining. Moreover, model training requires the processing on the blurryimage of first segmenting, then calculating respectively, and finallysynthesizing, which has a low model training efficiency.

Based on this, an embodiment of this application provides an imageprocessing model for processing blurry images. The image processingmodel may be used for performing sharpness restoration on a blurry imageto obtain a clear image.

The image processing model for processing blurry images provided In oneembodiment consistent with the present disclosure may be formed bysequentially connecting at least two networks with different scales inseries according to the scales in descending order or in ascendingorder, and the networks with different scales may perform sharpnessrestoration on blurry images with different sharpness. The scales of thenetworks are used for indicating the levels of the sharpness of imagesinputted to the networks, that is, the sizes of the scales correspond tothe measurements of the sharpness of to-be-processed blurry images. Acoarse scale represents that an original blurry image is downsampled toa lower resolution to obtain an image with higher sharpness, and a blurdegree of the image is relatively low. A fine scale represents that theoriginal blurry image is downsampled to a higher resolution to obtain animage with lower sharpness, and the blur degree of the image isrelatively high. Referring to FIG. 1 (an example in which an imageprocessing model includes three networks is used for description in FIG.1 ), the image processing model may include such three networks withdifferent scales as a coarse-scale network 11, a middle-scale network12, and a fine-scale network 13. In the three networks with differentscales, the scale of the coarse-scale network 11 is the largest, thescale of the middle-scale network 12 is the second largest, and thescale of the fine-scale network 13 is the smallest. Each network is ancodec network, which may specifically include a plurality of featureextraction units 111 (as shown in black units in FIG. 1 ), a pluralityof feature transformation units 112 (as shown in gray units in FIG. 1 ),and a plurality of feature reconstruction units 113 (as shown in whiteunits in FIG. 1 ) that have different channel quantities. The channelquantity of each unit may be set according to an empirical value orservice requirements, for example, setting to 32 channels, 64 channels,128 channels, and the like. In some embodiments, each of the featureextraction units 111, the feature transformation units 112, and thefeature reconstruction units 113 may include one or more convolutionallayers, and each convolutional layer may include two convolution kernelsof 3×3. The parameter quantity may be reduced by using the twoconvolution kernels of 3×3 to improve the speed of model training. FIG.1 only schematically represents a structure of the image processingmodel, and does not limit the structure of the image processing modelprovided In one embodiment consistent with the present disclosure. In anexample, the network quantity in the image processing model is notlimited to 3 shown in FIG. 1 , but may alternatively be 2, 4, or thelike. In another example, each convolutional layer may alternativelyinclude three convolution kernels of 3×3 or one convolution kernels of5×5.

Network parameters of the feature extraction units 111 in any twonetworks are independent, and/or network parameters of the featurereconstruction units 113 in any two networks are independent. Networkparameters of the feature transformation units 112 in any two networksare shared. In some embodiments, the feature transformation unit 112 mayinclude at least two residual units, each residual unit may include twoor more convolutional layers, and the residual units may be connected toeach other by using a multi-order nested skip connection structure. Aresidual unit may be defined with reference to formula 1.1:x _(n) =x _(n-1) +F _(n)(x _(n-1))  Formula 1.1

where, x_(n-1), x_(n), and F_(n) represent an input, an output, and aresidual function of an n^(th) residual unit; and formula 1.1 may alsobe referred to as a first-order residual function corresponding to astructure shown in FIG. 2A. In one embodiment consistent with thepresent disclosure, assuming that an input of an (n−1)^(th) anotherresidual unit is also generated by another residual function, the inputis substituted into formula 1.1 to obtain a second-order residualfunction shown in formula 1.2 corresponding to a second-order nestedskip connection structure shown in a schematic structural diagram on theleft side in FIG. 2B. In addition, it may be learned by comparingschematic structural diagrams on the left and right sides in FIG. 2Bthat, compared with directly connecting two residual units in series inthe related art, the second-order nested skip connection structureprovided In one embodiment consistent with the present disclosure hasone more connection.x _(n) =x _(n-2) +F _(n-1)(x _(n-2))+F _(n)(x _(n-2) +F _(n-1)(x_(n-2)))  Formula 1.2

Formula 1.2 is expanded to further obtain a third-order residualfunction shown in formula 1.3 corresponding to a third-order nested skipconnection structure shown in FIG. 2C.x _(n) =x _(n-3) +F _(n-2)(x _(n-3))+(x _(n-3) +F _(n-2)(x _(n-3)))+F_(n)(x _(n-3) +F _(n-2)(x _(n-3))+(x _(n-3) +F _(n-2)(x_(n-3))))  Formula 1.3

Similarly, a multi-order residual function and a correspondingmulti-order nested skip connection structure may be obtained. Themulti-order nested skip connection structure may be combined into anested module to be embedded in the feature transformation unit, therebyimproving gradient propagation and reducing the complexity of networkparameter optimization. Taking an example in which the featuretransformation unit 112 includes four residual units Fi, each residualunit includes two convolutional layers, and the residual units areconnected to each other by using a fourth-order nested skip connectionstructure, a schematic structural diagram of the feature transformationunit 112 may be shown with reference to FIG. 2D. FIG. 2D is theschematic diagram of the internal structure of the featuretransformation unit 112 in FIG. 1 . FIG. 2D only schematicallyrepresents a structure of the feature transformation unit 112, and doesnot limit the structure of the feature transformation unit 112 providedIn one embodiment consistent with the present disclosure. In an example,the quantity of the residual units in the feature transformation unit112 is not limited to 4 shown in FIG. 2D, but may alternatively be 2, 5,or the like. In another example, each residual unit is not limited toonly including two convolutional layers, but may alternatively includethree, five, or more convolutional layers.

For the foregoing image processing model, an embodiment of thisapplication further provides a model training solution to better trainand update the image processing model, optimize the deblurringperformance of the image processing model, and improve the efficiency ofmodel training. When the model training solution is used for trainingand updating the image processing model, a blurry image and a clearimage that are paired may be obtained; networks in the image processingmodel are sequentially called to perform sharpness restoration on theblurry image for training to obtain a restored image; and then networkparameters of the networks in the image processing model are updatedaccording to the restored image and the clear image. When each networkperforms sharpness restoration on an image, a plurality of encodingstages and a plurality of decoding stages may be included (threeencoding stages and three decoding stages are used as an example fordescription in FIG. 1 ). In each encoding stage, the feature extractionunit 111 may be first called to perform feature extraction on a receivedimage, and then the feature transformation unit 112 is called to performfeature transformation on an image obtained after the featureextraction. In each decoding stage, the feature transformation unit 112may be first called to perform feature transformation on a receivedimage, and then the feature reconstruction unit 113 is called to performfeature reconstruction on an image obtained after the featuretransformation.

Based on the foregoing description, the embodiments of this applicationprovide a training method and apparatus for an image processing modelfor processing blurry images, a network device, and a storage medium.

The training apparatus for an image processing model may be specificallyintegrated into the network device such as a terminal or a server. Theterminal herein may include, but is not limited to: a smart terminal, atablet computer, a laptop computer, a desktop computer, or the like. Forexample, referring to FIG. 3A, a network device 31 may obtain a samplepair for training, the sample pair including a clear image and a blurryimage corresponding to the clear image; and the sharpness of the clearimage being greater than a preset threshold, and the sharpness of theblurry image being less than the preset threshold; activate the imageprocessing model to perform sharpness restoration on the blurry image toobtain a restored image; and update network parameters of the firstnetwork and/or network parameters of the second network in the imageprocessing model according to the restored image and the clear image toobtain a trained image processing model.

A training method for an image processing model for processing blurryimages provided in an embodiment of this application may be performed bya network device. Referring to FIG. 3B, the training method for an imageprocessing model for processing blurry images may include the followingsteps S301 to S303:

S301. Obtain a sample pair for training, the sample pair including aclear image and a blurry image corresponding to the clear image.

When the sample pair for training is obtained, the clear image and theblurry image corresponding to the clear image may be obtained in adata-driven manner. The so-called data-driven manner refers to a mannerof blurring a dynamic scene by superimposing a plurality of consecutiveframes of images captured by a camera to obtain the blurry image and theclear image in the dynamic scene. The sharpness of the clear image isgreater than a preset threshold, and the sharpness of the blurry imageis less than the preset threshold. The preset threshold herein may beset according to an empirical value or actual service requirements (forexample, the requirement on the accuracy of deblurring performance ofthe image processing model). The clear image and the blurry image thatare paired are obtained in the data-driven manner, which may reduce theacquisition difficulty of the sample pair.

S302. Activate the image processing model to perform sharpnessrestoration on the blurry image to obtain a restored image.

In one embodiment consistent with the present disclosure, the imageprocessing model at least includes a first network and a second network;and the first network and the second network are codec networks withdifferent scales, the first network corresponds to a first scale, andthe second network corresponds to a second scale. Values of the firstscale and the second scale are different, and the value of the firstscale may be greater than the value of the second scale, that is, thefirst scale may be a coarse scale, and the second scale may be a finescale. When the image processing model is called to perform sharpnessrestoration on the blurry image, the first network and the secondnetwork may be sequentially called to perform sharpness restoration onthe blurry image to obtain the restored image. If the image processingmodel further includes other networks such as a third network and afourth network, the first network, the second network, and the othernetworks may be called to perform sharpness restoration on the blurryimage.

The sharpness restoration herein refers to the processing of improvingthe sharpness of the image. FIG. 3C is a flowchart of sharpnessrestoration according to an embodiment of this application. In someembodiments, as shown in FIG. 3C, the following steps S321 to S323 arespecifically included:

S321. Perform feature extraction on an image.

Specifically, a plurality of convolution operations may be performed onthe image to implement the feature extraction on the image, or a featureextraction algorithm may be used to perform the feature extraction onthe image. The feature extraction algorithm herein may include, but isnot limited to: a local binary patterns (LBP) algorithm, a histogram oforiented gradient (HOG) feature extraction algorithm, a speeded uprobust features (SURF) algorithm, or the like.

S322. Perform, by using a multi-order residual function, featuretransformation on an image obtained after the feature extraction.

The multi-order residual function herein refers to a residual functionof which an order is greater than or equal to 2.

S323. Perform feature reconstruction on an image obtained after thefeature transformation.

Specifically, a plurality of deconvolution operations may be performedon the image obtained after the feature transformation to implement thefeature reconstruction on the image obtained after the featuretransformation.

S303. Update network parameters of the first network and/or networkparameters of the second network in the image processing model accordingto the restored image and the clear image to obtain a trained imageprocessing model.

Through the study of blurry images (images shown on the left side inFIG. 4 ) captured in a dynamic scene, it is found that, in the imagesshown on the left side in FIG. 4 , an image of a building part in abackground region is relatively clear, while an image of a crowd part ina foreground region is relatively blurry. A blurry image region 21 inthe foreground region and a clear image region 22 in the backgroundregion are randomly selected, and the two selected image regions areanalyzed in an image pyramid. For an analysis result thereof, refer tothe right side in FIG. 4 . According to the analysis result shown on theright side in FIG. 4 , it may be learned that, after the image region 22in the background region is downsampled, edges of the image thereof arestill clear after downsampling; and after the image region 21 in theforeground region is downsampled, edges of the image thereof becomeincreasingly clear after downsampling. If the same feature extractionparameters are allocated to the networks with different scales in theimage processing model, the image processing model cannot both extractclear image features and blurry image features. Therefore, in anembodiment of this application, different feature extraction parametersare allocated to feature extraction units of the networks with differentscales, which enables the networks with different scales to learnimportant image information at the different scales, so as to extractmore image features at the different scales.

Because feature transformation functions of the feature transformationunits in the networks with different scales are similar, and all aim totransform corresponding blurry image features into clear image features,in an embodiment of this application, the same feature transformationparameters are allocated to feature transformation units of the networkswith different scales, as shown in FIG. 5A. Three rows in FIG. 5Arepresent the three networks at the coarse scale to the fine scalerespectively from top to bottom. FE represents a feature extractionunit, T represents a feature transformation unit, and the samebackground represents the same parameters. Further, because functions ofthe feature transformation at different scales and the same scale aresimilar, the same feature transformation parameters may be furtherallocated to the feature transformation units in the networks with thesame scale, as shown in FIG. 5B. FIG. 5A and FIG. 5B only schematicallyrepresent encoding stages of the networks, and decoding stages thereofare not shown in FIG. 5A and FIG. 5B.

Based on the foregoing description, for the first network and the secondnetwork in the image processing model, the network parameters of thefirst network and the network parameters of the second network meeting aselective sharing condition may be set, and the selective sharingcondition is used for indicating shared network parameters between thefirst network and the second network, and is used for indicatingindependent network parameters between the first network and the secondnetwork. Specifically, the network parameters include a featureextraction parameter and a feature transformation parameter; theselective sharing condition, when being used for indicating the sharednetwork parameters between the first network and the second network, isspecifically used for indicating that the feature transformationparameter of the first network and the feature transformation parameterof the second network are the shared network parameters, that is, thefeature transformation parameter of the first network and the featuretransformation parameter of the second network are the same networkparameter; and the selective sharing condition, when being used forindicating the independent network parameters between the first networkand the second network, is specifically used for indicating that thefeature extraction parameter of the first network and the featureextraction parameter of the second network are the independent networkparameters, that is, the feature extraction parameter of the firstnetwork and the feature extraction parameter of the second network aredifferent network parameters. In some embodiments, the networkparameters further include a feature reconstruction parameter; and theselective sharing condition, when being used for indicating theindependent network parameters between the first network and the secondnetwork, is further used for indicating that the feature reconstructionparameter of the first network and the feature reconstruction parameterof the second network are the independent network parameters, that is,the feature reconstruction parameter of the first network and thefeature reconstruction parameter of the second network are differentnetwork parameters.

The selective sharing condition being specifically used for indicatingthat the feature transformation parameter of the first network and thefeature transformation parameter of the second network are the sharednetwork parameters may include the following two implementations: (1)when the quantity of the feature transformation parameter is greaterthan 1, a plurality of feature transformation parameters of the firstnetwork and a plurality of feature transformation parameters of thesecond network are the shared network parameters, and each of thefeature transformation parameters of the first network is an independentnetwork parameter and each of the feature transformation parameters ofthe second network is an independent network parameter, as shown in theimage on the right side in FIG. 5A; and (2) when the quantity of thefeature transformation parameter is greater than 1, a plurality offeature transformation parameters of the first network and a pluralityof feature transformation parameters of the second network are theshared network parameters, and each of the feature transformationparameters of the first network is a shared network parameter and eachof the feature transformation parameters of the second network is ashared network parameter, as shown in the image on the right side inFIG. 5B.

In one embodiment consistent with the present disclosure, the imageprocessing model at least includes the first network with the firstscale and the second network with the second scale. Because there areshared network parameters and independent network parameters between thefirst network and the second network, when performing sharpnessrestoration on the blurry image, the image processing model can learnmore image features in the blurry image to obtain a more accuraterestored image. The network parameters of the first network and/or thenetwork parameters of the second network are updated according to themore accurate restored image and the clear image, which may improve thedeblurring performance of the trained image processing model. Inaddition, because there are shared network parameters between the firstnetwork and the second network, the quantity of parameters of the imageprocessing model may be reduced, and the efficiency of model training isimproved. Moreover, by using the corresponding clear image and blurryimage to perform end-to-end training and learning on the imageprocessing model, there is no need to segment the blurry image intoblurry regions in movement, and there is no need to make any assumptionon the blurry image, which may further improve the deblurringperformance of the trained image processing model and the efficiency ofmodel training.

FIG. 6A is a schematic flowchart of another training method for an imageprocessing model for processing blurry images according to an embodimentof this application. The training method for an image processing modelmay be performed by a network device. Referring to FIG. 6A, the trainingmethod for an image processing model may include the following stepsS601 to S605:

S601. Obtain a sample pair for training, the sample pair including aclear image and a blurry image corresponding to the clear image.

The network device may obtain a large quantity of sample pairs andperform a subsequent model training update operation on the imageprocessing model by using the large quantity of sample pairs. In oneembodiment, because the production of a blurry image is usually causedby camera movement during recording or object movement in a recordingscene, and is essentially because a shutter speed of a camera is notfast enough. As a result, the camera movement or the object movement inthe recording scene causes a sensor of the camera to acquire not onlythe luminance at a certain fixed location, but also an integral of allluminance at related locations within a period of time in which ashutter is enabled and then disabled, resulting in image blurring.However, studies show that the integral of all luminance at relatedlocations in consecutive frames of images captured by the camera may beapproximately the summation of adjacent consecutive images.

FIG. 6B is a flowchart of a method for obtaining a sample pair fortraining according to an embodiment of this application. As shown inFIG. 6 , the method for obtaining a sample pair for training mayspecifically include the following steps S611 to S613:

Step S611. Obtain image sequence frames for training.

In some embodiments, the image sequence frames may be obtained byacquiring, by using an action camera (for example, a GoPro high-speedcamera) and a high-speed mode of a network device, a large number ofvideos, and performing image frame analysis on the acquired videos. Thevideos may be high-speed videos at 240 frames per second, high-speedvideos at 120 frames per second, or the like.

Step S612. Select or randomly select one frame of image from the imagesequence frames as a clear image, and determine a plurality of frames ofreference images associated with the clear image.

In some embodiments, the reference image being associated with the clearimage refers to: a difference between a frame sequence number of thereference image and a frame sequence number of the clear image beingless than a preset difference. For example, a frame sequence number ofthe clear image is 5, that is, the clear image is a 5^(th) frame ofimage in the image sequence frames. If the preset difference is 3, a3^(rd) frame of image, a 4^(th) frame of image, a 6^(th) frame of image,and a 7^(th) frame of image in the image sequence frames may all be usedas the reference images.

Step S613. Obtain a blurry image corresponding to the clear imageaccording to the plurality of frames of reference images, and construct,by using the blurry image and the clear image, the sample pair fortraining.

In some embodiments, an implementation of the obtaining a blurry imagecorresponding to the clear image according to the plurality of frames ofreference images may be: superimposing and averaging the plurality offrames of reference images to obtain the blurry image corresponding tothe clear image.

S602. Activate the image processing model to perform sharpnessrestoration on the blurry image to obtain a restored image.

In one embodiment consistent with the present disclosure, the imageprocessing model at least includes a first network and a second network;the first network corresponds to a first scale, and the second networkcorresponds to a second scale; and values of the first scale and thesecond scale are different. As can be learned from the foregoing, thenetworks with different scales may perform sharpness restoration onblurry images with different sharpness.

FIG. 6C is a flowchart of a method for activating an image processingmodel to perform sharpness restoration on a blurry image in step S602according to an embodiment of this application. As shown in FIG. 6C, themethod includes the following steps S621 to S624:

Step S621. Downsample the blurry image according to the first scale toobtain a blurry image with first sharpness.

Step S622. Activate the first network to perform sharpness restorationon the blurry image with the first sharpness to obtain an intermediateimage.

In some embodiments, the first network may perform sharpness restorationon the blurry image with the first sharpness by using formula 1.4.I ₁ =Net ₁(B ₁;θ₁,η)  Formula 1.4

where, Net₁ is a function used by the first network to perform sharpnessrestoration, B₁ represents the blurry image with the first sharpnessinputted to the first network, θ₁ represents a network parameter,independent of the second network, in the first network, η represents anetwork parameter shared between the first network and the secondnetwork, and I₁ represents the intermediate image outputted by the firstnetwork.

Step S623. Downsample the blurry image according to the second scale toobtain a blurry image with second sharpness.

Step S624. Activate the second network to perform sharpness restorationaccording to the blurry image with the second sharpness and theintermediate image to obtain a restored image.

In some embodiments, the second network may perform sharpnessrestoration on the blurry image with the second sharpness and theintermediate image by using formula 1.5.I ₂ =Net ₂(B ₂ ,I ₁;θ₂,η)  Formula 1.5

where, Net₂ is a function used by the second network to performsharpness restoration, B₂ represents the blurry image with the secondsharpness inputted to the second network, I₁ represents the intermediateimage outputted by the first network, θ₂ represents a network parameter,independent of the first network, in the second network, η represents anetwork parameter shared between the first network and the secondnetwork, and I₂ represents the restored image outputted by the secondnetwork.

When the image processing model includes at least three networks, thenetworks may be sequentially called to perform sharpness restoration onthe blurry image according to a connection order of the image processingmodel. A first network in the image processing model may performsharpness restoration by using formula 1.4, any network other than thefirst network in the image processing model may perform sharpnessrestoration by using formula 1.6, and an image obtained by performingsharpness restoration by a last network is a restored image.I _(i) =Net _(i)(B _(i) ,I _(i-1);θ_(i),η)  Formula 1.6

where, Net_(i) is a function used by an i^(th) network to performsharpness restoration, B_(i) represents a blurry image with i^(th)sharpness inputted to the i^(th) network, θ₂ represents a networkparameter, independent of other networks with different scales, in thei^(th) network, η represents a network parameter shared between thenetworks, I_(i-1) represents an intermediate image outputted by an(i−1)^(th) network, and I_(i) represents an intermediate image outputtedby the i^(th) network.

S603. Obtain an optimization function of the image processing model.

S604. Determine a value of the optimization function according to therestored image and the clear image.

S605. Update, by reducing the value of the optimization function, thenetwork parameters of the first network and/or the network parameters ofthe second network in the image processing model, to obtain a trainedimage processing model.

The network device may perform the foregoing steps S601 and S602 toobtain restored images and clear images of a large quantity of samplepairs, and perform steps S603 to S605 after obtaining a large quantityof pairs of restored images and clear images. In steps S603 to S605, theoptimization function of the image processing model may be shown asformula 1.2:

$\begin{matrix}{{f\left( {\theta,\eta} \right)} = {\frac{1}{2N}{\sum\limits_{k = 1}^{N}{\sum\limits_{i = 1}^{S}{\frac{1}{T_{i}}{{{F_{i}\left( {{B_{i}^{k};\theta_{i}},\eta} \right)} - L_{i}^{k}}}_{2}^{2}}}}}} & {{Formula}1.2}\end{matrix}$

where, N represents the quantity of the sample pairs, B_(i) ^(k) andL_(i) ^(k) respectively represent the blurry image and the clear imagein a k^(th) sample pair at scale i, S represents the total quantity ofscales in the image processing model, θ_(i) represents an independentnetwork parameter in a network corresponding to the scale i, ηrepresents a shared network parameter, T_(i) represents the totalquantity of pixels of the image at the scale i, F_(i) and represents afunction for performing sharpness restoration on the blurry image B_(i)^(k).

After the optimization function is obtained, the restored image and theclear image may be substituted into the optimization function todetermine the value of the optimization function, and then the networkparameters of the first network and/or the network parameters of thesecond network in the image processing model are continuously updated byreducing the value of the optimization function, until the value of theoptimization function is minimized, and the image processing model is ina converged state. The image processing model may further include othernetworks different from the first network and the second network. Then,after the value of the optimization function is determined, networkparameters of the other networks in the image processing model may befurther continuously updated by reducing the value of the optimizationfunction.

In one embodiment consistent with the present disclosure, the imageprocessing model for processing blurry images at least includes thefirst network with the first scale and the second network with thesecond scale. Because there are shared network parameters andindependent network parameters between the first network and the secondnetwork, when performing sharpness restoration on the blurry image, theimage processing model can learn more image features in the blurry imageto obtain a more accurate restored image. The network parameters of thefirst network and/or the network parameters of the second network areupdated according to the more accurate restored image and the clearimage, which may improve the deblurring performance of the trained imageprocessing model. In addition, because there are shared networkparameters between the first network and the second network, thequantity of parameters of the image processing model may be reduced, andthe efficiency of model training is improved. Moreover, by using thecorresponding clear image and blurry image to perform end-to-endtraining and learning on the image processing model, there is no need tosegment the blurry image into blurry regions in movement, and there isno need to make any assumption on the blurry image, which may furtherimprove the deblurring performance of the trained image processing modeland the efficiency of model training.

Based on the foregoing related description of the image processingmodel, an embodiment of this application further provides an imageprocessing method, and the image processing method may be performed bythe network device in FIG. 3A. Referring to FIG. 7 , the imageprocessing method may include the following steps S701 to S703:

S701. Obtain a to-be-processed original image.

The sharpness of the original image is less than a preset threshold. Ato-be-processed original image may be obtained using the following twomethods:

(1) Actively obtain a to-be-processed original image.

Specifically, when a camera assembly is used for image capture, if thenetwork device detects that the camera assembly is in a dynamic scene ora relatively dark environment, whether the sharpness of the imagecaptured by the camera assembly is less than the preset threshold may beactively detected; and if the sharpness is less than the presetthreshold, the network device may actively use the image captured by thecamera assembly as the to-be-processed original image. In one example,when the camera assembly of the network device records an environmentalcondition in a certain region, if the network device determines thatthere is usually a flow of people or vehicles in the region according tohistorical environmental data of the region, it may be considered thatthe camera assembly is in the dynamic environment. In this case, whetherthe sharpness of the image captured by the camera assembly is less thanthe preset threshold may be actively detected; and if the sharpness isless than the preset threshold, the network device may actively use theimage captured by the camera assembly as the to-be-processed originalimage. In another example, when the camera assembly of the networkdevice captures an image, if the network device detects that the lightvalue of the environment in which the camera assembly is located is lessthan a preset light value according to a light sensor or the cameraassembly, it may be considered that the camera assembly is in therelatively dark environment. In this case, whether the sharpness of theimage captured by the camera assembly is less than the preset thresholdmay be actively detected; and if the sharpness is less than the presetthreshold, the network device may actively use the image captured by thecamera assembly as the to-be-processed original image.

(2) Obtain a to-be-processed original image according to a userinstruction.

In one embodiment, after the network device detects that the user usesthe camera assembly of the network device to capture an image, the imagecaptured by the camera assembly may be obtained, and the captured imageis displayed in a user interface, for the user to view. If finding thatthe captured image is not clear and the sharpness thereof is less thanthe preset threshold, the user may input an image processing instructionto the network device. If the network device receives the imageprocessing instruction, the captured image may be used as theto-be-processed original image. In another embodiment, if the user findsthat some historical images in an image gallery of the network deviceare blurry and the sharpness thereof is less than the preset threshold,the user may also input an image processing instruction to the networkdevice to trigger the network device to obtain the historical images asthe to-be-processed original images. The foregoing image processinginstruction may be an instruction generated by the user by clicking orpressing an image, or may be an instruction generated by the user bypressing a designated key on the network device, or may be aninstruction generated by the user by inputting voice to the networkdevice, or the like.

S702. Activate the image processing model to perform sharpnessrestoration on the original image to obtain a target image.

The sharpness of the target image is greater than the preset threshold.The sharpness restoration includes: performing feature extraction on animage, performing, by using a multi-order residual function, featuretransformation on an image obtained after the feature extraction, andperforming feature reconstruction on an image obtained after the featuretransformation. Correspondingly, in an implementation process of stepS702, the image processing model may be called to first perform featureextraction on the original image to obtain a first image obtained afterthe feature extraction; feature transformation is then performed on thefirst image by using a multi-order residual function to obtain a secondimage obtained after the feature transformation; and finally featurereconstruction is performed on the second image to obtain the targetimage.

The image processing model herein may be obtained by training by usingthe training method for an image processing model shown in FIG. 3B orFIG. 6A. The image processing model at least includes a first networkand a second network; the first network corresponds to a first scale,and the second network corresponds to a second scale; and networkparameters of the first network and network parameters of the secondnetwork meet a selective sharing condition, and the selective sharingcondition is used for indicating shared network parameters between thefirst network and the second network, and is used for indicatingindependent network parameters between the first network and the secondnetwork. In one embodiment, the network parameters include a featureextraction parameter and a feature transformation parameter.Correspondingly, the selective sharing condition, when being used forindicating the shared network parameters between the first network andthe second network, is specifically used for indicating that the featuretransformation parameter of the first network and the featuretransformation parameter of the second network are the shared networkparameters; and the selective sharing condition, when being used forindicating the independent network parameters between the first networkand the second network, is specifically used for indicating that thefeature extraction parameter of the first network and the featureextraction parameter of the second network are the independent networkparameters. In another embodiment, the network parameters furtherinclude a feature reconstruction parameter. Correspondingly, theselective sharing condition, when being used for indicating theindependent network parameters between the first network and the secondnetwork, is further used for indicating that the feature reconstructionparameter of the first network and the feature reconstruction parameterof the second network are the independent network parameters.

S703. Output the target image.

In one embodiment consistent with the present disclosure, because theimage processing model is obtained by training by using the trainingmethod for an image processing model shown in FIG. 3B or FIG. 6A, thedeblurring performance of the image processing model is good. Therefore,by activating the image processing model to perform sharpnessrestoration on the original image with low sharpness, the original imagemay be better deblurred to obtain a relatively clear target image, whichmay improve the sharpness of the target image and further improve theimage quality of the target image.

Based on the foregoing description of the embodiment of the trainingmethod for an image processing model, an embodiment of this applicationfurther discloses a training apparatus for an image processing model forprocessing blurry images. The image processing model at least includes afirst network and a second network; the first network and the secondnetwork are codec networks with different scales; the sizes of thescales correspond to the measurements of the sharpness ofto-be-processed blurry images; and the training apparatus for an imageprocessing model may be a computer program (including program code) runon a network device. The training apparatus for an image processingmodel may perform the method shown in FIG. 3B or FIG. 6A. Referring toFIG. 8 , the training apparatus for an image processing model mayoperate the following units:

an obtaining unit 101, configured to obtain a sample pair for training,the sample pair including a clear image and a blurry image correspondingto the clear image; and the sharpness of the clear image being greaterthan a preset threshold, and the sharpness of the blurry image beingless than the preset threshold;

a processing unit 102, configured to activate the image processing modelto perform sharpness restoration on the blurry image to obtain arestored image; and

an update unit 103, configured to update network parameters of the firstnetwork and/or network parameters of the second network in the imageprocessing model according to the restored image and the clear image toobtain a trained image processing model;

the network parameters of the first network and the network parametersof the second network meeting a selective sharing condition, and theselective sharing condition being used for indicating shared networkparameters between the first network and the second network, and beingused for indicating independent network parameters between the firstnetwork and the second network.

In one embodiment, the network parameters include a feature extractionparameter and a feature transformation parameter;

the selective sharing condition, when being used for indicating theshared network parameters between the first network and the secondnetwork, is specifically used for indicating that the featuretransformation parameter of the first network and the featuretransformation parameter of the second network are the shared networkparameters; and

the selective sharing condition, when being used for indicating theindependent network parameters between the first network and the secondnetwork, is specifically used for indicating that the feature extractionparameter of the first network and the feature extraction parameter ofthe second network are the independent network parameters.

In another embodiment, the network parameters further include a featurereconstruction parameter; and

the selective sharing condition, when being used for indicating theindependent network parameters between the first network and the secondnetwork, is further used for indicating that the feature reconstructionparameter of the first network and the feature reconstruction parameterof the second network are the independent network parameters.

In another embodiment, the selective sharing condition beingspecifically used for indicating that the feature transformationparameter of the first network and the feature transformation parameterof the second network are the shared network parameters includes:

when the quantity of the feature transformation parameter is greaterthan 1, a plurality of feature transformation parameters of the firstnetwork and a plurality of feature transformation parameters of thesecond network being the shared network parameters, and each of thefeature transformation parameters of the first network being anindependent network parameter and each of the feature transformationparameters of the second network being an independent network parameter;or

when the quantity of the feature transformation parameter is greaterthan 1, a plurality of feature transformation parameters of the firstnetwork and a plurality of feature transformation parameters of thesecond network being the shared network parameters, and each of thefeature transformation parameters of the first network being a sharednetwork parameter and each of the feature transformation parameters ofthe second network being a shared network parameter.

In another embodiment, the first network corresponds to a first scale,and the second network corresponds to a second scale; and the processingunit 102, when being configured to activate the image processing modelto perform sharpness restoration on the blurry image to obtain arestored image, is specifically configured to:

downsample the blurry image according to the first scale to obtain ablurry image with first sharpness;

activate the first network to perform sharpness restoration on theblurry image with the first sharpness to obtain an intermediate image;

downsample the blurry image according to the second scale to obtain ablurry image with second sharpness; and

activate the second network to perform sharpness restoration accordingto the blurry image with the second sharpness and the intermediate imageto obtain a restored image.

In another embodiment, the sharpness restoration includes: performingfeature extraction on an image, performing, by using a multi-orderresidual function, feature transformation on an image obtained after thefeature extraction, and performing feature reconstruction on an imageobtained after the feature transformation.

In another embodiment, the update unit 103, when being configured toupdate network parameters of the first network and/or network parametersof the second network in the image processing model according to therestored image and the clear image, is specifically configured to:

obtain an optimization function of the image processing model;

determine a value of the optimization function according to the restoredimage and the clear image; and

update, according to the principle of reducing the value of theoptimization function, the network parameters of the first networkand/or the network parameters of the second network in the imageprocessing model.

In another embodiment, the obtaining unit 101, when being configured toobtain a sample pair for training, is specifically configured to:

obtain image sequence frames for training, the image sequence framesincluding at least two frames of images;

select or randomly select one frame of image from the image sequenceframes as a clear image, and determine a plurality of frames ofreference images associated with the clear image; and

obtain a blurry image corresponding to the clear image according to theplurality of frames of reference images, and construct, by using theblurry image and the clear image, the sample pair for training.

In another embodiment, the obtaining unit 101, when being configured toobtain a blurry image corresponding to the clear image according to theplurality of frames of reference images, is specifically configured to:

superimpose and average the plurality of frames of reference images toobtain the blurry image corresponding to the clear image.

According to an embodiment of this application, the steps in the methodshown in FIG. 3B or FIG. 6A may be performed by the units of thetraining apparatus for an image processing model shown in FIG. 8 . Inone example, steps S301 to S303 shown in FIG. 3B may be respectivelyperformed by the obtaining unit 101, the processing unit 102, and theupdate unit 103 shown in FIG. 8 . In another example, steps S601 andS602 shown in FIG. 6A may be respectively performed by the obtainingunit 101 and the processing unit 102 shown in FIG. 8 , and steps S603 toS605 may be performed by the update unit 103 shown in FIG. 8 .

According to another embodiment of this application, the units of thetraining apparatus for an image processing model for processing blurryimages shown in FIG. 8 may be separately or wholly combined into one orseveral other units, or one (or more) of the units herein may further bedivided into a plurality of units of smaller functions. In this way,same operations may be implemented, and the implementation of thetechnical effects of the embodiments of this application is notaffected. The foregoing units are divided based on logical functions. Inan actual application, a function of one unit may also be implemented bya plurality of units, or functions of a plurality of units areimplemented by one unit. In other embodiments of this application, thetraining apparatus for an image processing model may also include otherunits. In an actual application, the functions may also be cooperativelyimplemented by other units and may be cooperatively implemented by aplurality of units.

According to another embodiment of this application, a computer program(including program code) that can perform the steps in the correspondingmethod shown in FIG. 3B or FIG. 6A may be run on a general computingdevice, such as a computer, which includes processing elements andstorage elements such as a central processing unit (CPU), a randomaccess memory (RAM), and a read-only memory (ROM), to construct thetraining apparatus for an image processing model shown in FIG. 8 andimplement the training method for an image processing model in theembodiments of this application. The computer program may be recordedon, for example, a computer-readable recording medium, and may be loadedinto the foregoing computing device by using the computer-readablerecording medium and run on the computing device.

In one embodiment consistent with the present disclosure, the imageprocessing model for processing blurry images at least includes thefirst network with the first scale and the second network with thesecond scale. Because there are shared network parameters andindependent network parameters between the first network and the secondnetwork, when performing sharpness restoration on the blurry image, theimage processing model can learn more image features in the blurry imageto obtain a more accurate restored image. The network parameters of thefirst network and/or the network parameters of the second network areupdated according to the more accurate restored image and the clearimage, which may improve the deblurring performance of the trained imageprocessing model. In addition, because there are shared networkparameters between the first network and the second network, thequantity of parameters of the image processing model may be reduced, andthe efficiency of model training is improved. Moreover, by using thecorresponding clear image and blurry image to perform end-to-endtraining and learning on the image processing model, there is no need tosegment the blurry image into blurry regions in movement, and there isno need to make any assumption on the blurry image, which may furtherimprove the deblurring performance of the trained image processing modeland the efficiency of model training.

Based on the foregoing description of the embodiment of the imageprocessing method, an embodiment of this application further disclosesan image processing apparatus. The image processing apparatus may be acomputer program (including program code) run on a network device. Theimage processing apparatus may perform the method shown in FIG. 7 .Referring to FIG. 9 , the image processing apparatus may operate thefollowing units:

an obtaining unit 201, configured to obtain a to-be-processed originalimage, the sharpness of the original image being less than a presetthreshold;

a processing unit 202, configured to activate an image processing modelfor processing blurry images to perform sharpness restoration on theoriginal image to obtain a target image, the sharpness of the targetimage being greater than the preset threshold, the image processingmodel at least including a first network and a second network; the firstnetwork and the second network being codec networks with differentscales; the sizes of the scales corresponding to the measurements of thesharpness of to-be-processed blurry images; and the network parametersof the first network and the network parameters of the second networkmeeting a selective sharing condition, and the selective sharingcondition being used for indicating shared network parameters betweenthe first network and the second network, and being used for indicatingindependent network parameters between the first network and the secondnetwork; and

an output unit 203, configured to output the target image.

In one embodiment, the network parameters include a feature extractionparameter and a feature transformation parameter;

the selective sharing condition, when being used for indicating theshared network parameters between the first network and the secondnetwork, is specifically used for indicating that the featuretransformation parameter of the first network and the featuretransformation parameter of the second network are the shared networkparameters; and

the selective sharing condition, when being used for indicating theindependent network parameters between the first network and the secondnetwork, is specifically used for indicating that the feature extractionparameter of the first network and the feature extraction parameter ofthe second network are the independent network parameters.

In another embodiment, the network parameters further include a featurereconstruction parameter; and

the selective sharing condition, when being used for indicating theindependent network parameters between the first network and the secondnetwork, is further used for indicating that the feature reconstructionparameter of the first network and the feature reconstruction parameterof the second network are the independent network parameters.

According to an embodiment of this application, the steps in the methodshown in FIG. 7 may be performed by the units of the image processingapparatus shown in FIG. 9 . Specifically, steps S701 to S703 shown inFIG. 7 may be respectively performed by the obtaining unit 201, theprocessing unit 202, and the output unit 203 shown in FIG. 9 . Accordingto another embodiment of this application, the units of the imageprocessing apparatus shown in FIG. 9 may be separately or whollycombined into one or several other units, or one (or more) of the unitsherein may further be divided into a plurality of units of smallerfunctions. In this way, same operations may be implemented, and theimplementation of the technical effects of the embodiments of thisapplication is not affected. The foregoing units are divided based onlogical functions. In an actual application, a function of one unit mayalso be implemented by a plurality of units, or functions of a pluralityof units are implemented by one unit. In other embodiments of thisapplication, the image processing apparatus may also include otherunits. In an actual application, the functions may also be cooperativelyimplemented by other units and may be cooperatively implemented by aplurality of units. According to another embodiment of this application,a computer program (including program code) that can perform the stepsin the corresponding method shown in FIG. 7 may be run on a generalcomputing device, such as a computer, which includes processing elementsand storage elements such as a central processing unit (CPU), a randomaccess memory (RAM), and a read-only memory (ROM), to construct theimage processing apparatus shown in FIG. 9 and implement the imageprocessing method in the embodiments of this application. The computerprogram may be recorded on, for example, a computer-readable recordingmedium, and may be loaded into the foregoing computing device by usingthe computer-readable recording medium and run on the computing device.

In one embodiment consistent with the present disclosure, because theimage processing model is obtained by training by using the trainingmethod for an image processing model shown in FIG. 3B or FIG. 6A, thedeblurring performance of the image processing model is good. Therefore,by activating the image processing model to perform sharpnessrestoration on the original image with low sharpness, the original imagemay be better deblurred to obtain a relatively clear target image, whichmay improve the sharpness of the target image and further improve theimage quality of the target image.

Based on the descriptions of the foregoing method embodiments andapparatus embodiments, an embodiment of this application furtherprovides a network device. Referring to FIG. 10 , the network device atleast includes a processor 301, an input device 302, an output device303, and a computer storage medium 304. The input device 302 may furtherinclude a camera assembly, and the camera assembly may be configured toacquire images. The camera assembly may be an assembly configured on thenetwork device when the network device leaves the factory, or may be anexternal assembly connected to the network device. In some embodiments,the network device may be further connected to other devices to receiveimages transmitted by the other devices.

The computer storage medium 304 may be stored in a memory of the networkdevice. The computer storage medium 304 is configured to store acomputer program. The computer program includes program instructions.The processor 301 is configured to execute the program instructionsstored in the computer storage medium 304. The processor 301 (orreferred to as a central processing unit (CPU)) is a computing core anda control core of the network device, is suitable for implementing oneor more instructions, and is specifically suitable for loading andexecuting one or more instructions to implement a corresponding methodprocedure or a corresponding function. In an embodiment, the processor301 in the embodiments of this application may be configured to performa series of training on the image processing model for processing blurryimages, including: obtaining a sample pair for training, the sample pairincluding a clear image and a blurry image corresponding to the clearimage; and the sharpness of the clear image being greater than a presetthreshold, and the sharpness of the blurry image being less than thepreset threshold; activating the image processing model to performsharpness restoration on the blurry image to obtain a restored image;and updating network parameters of the first network and/or networkparameters of the second network in the image processing model accordingto the restored image and the clear image; the network parameters of thefirst network and the network parameters of the second network meeting aselective sharing condition, and the selective sharing condition beingused for indicating shared network parameters between the first networkand the second network, and being used for indicating independentnetwork parameters between the first network and the second network. Inanother embodiment, the processor 301 in the embodiments of thisapplication may be further configured to perform a series of imageprocessing on the original image, including: obtaining a to-be-processedoriginal image, the sharpness of the original image being less than apreset threshold; and activating the image processing model to performsharpness restoration on the original image to obtain a target image,the sharpness of the target image being greater than the presetthreshold.

An embodiment of this application further provides a computer storagemedium (memory), and the computer storage medium is a memory device inthe network device and is configured to store programs and data. It maybe understood that the computer storage medium herein may include aninternal storage medium in the network device and certainly may alsoinclude an extended storage medium supported by the network device. Thecomputer storage medium provides storage space, and the storage spacestores an operating system of the network device. In addition, thestorage space further stores one or more instructions suitable for beingloaded and executed by the processor 301. The instructions may be one ormore computer programs (including program code). The computer storagemedium herein may be a high-speed RAM or a non-volatile memory, forexample, at least one magnetic disk memory. In some embodiments, thecomputer storage medium may be at least one computer storage mediumlocated away from the foregoing processor.

In an embodiment, one or more first instructions stored in the computerstorage medium may be loaded and executed by the processor 301 toimplement corresponding steps of the method in the foregoing embodimentsrelated to training of the image processing model. The image processingmodel at least includes a first network and a second network; the firstnetwork and the second network are codec networks with different scales;and the sizes of the scales correspond to the measurements of thesharpness of to-be-processed blurry images. In an implementation, theone or more first instructions in the computer storage medium are loadedby the processor 301 to perform the following steps:

obtaining a sample pair for training, the sample pair including a clearimage and a blurry image corresponding to the clear image; and thesharpness of the clear image being greater than a preset threshold, andthe sharpness of the blurry image being less than the preset threshold;

activating the image processing model to perform sharpness restorationon the blurry image to obtain a restored image; and

updating network parameters of the first network and/or networkparameters of the second network in the image processing model accordingto the restored image and the clear image;

the network parameters of the first network and the network parametersof the second network meeting a selective sharing condition, and theselective sharing condition being used for indicating shared networkparameters between the first network and the second network, and beingused for indicating independent network parameters between the firstnetwork and the second network.

In one embodiment, the network parameters include a feature extractionparameter and a feature transformation parameter;

the selective sharing condition, when being used for indicating theshared network parameters between the first network and the secondnetwork, is specifically used for indicating that the featuretransformation parameter of the first network and the featuretransformation parameter of the second network are the shared networkparameters; and the selective sharing condition, when being used forindicating the independent network parameters between the first networkand the second network, is specifically used for indicating that thefeature extraction parameter of the first network and the featureextraction parameter of the second network are the independent networkparameters.

In another embodiment, the network parameters further include a featurereconstruction parameter; and

the selective sharing condition, when being used for indicating theindependent network parameters between the first network and the secondnetwork, is further used for indicating that the feature reconstructionparameter of the first network and the feature reconstruction parameterof the second network are the independent network parameters.

In another embodiment, the selective sharing condition beingspecifically used for indicating that the feature transformationparameter of the first network and the feature transformation parameterof the second network are the shared network parameters includes:

when the quantity of the feature transformation parameter is greaterthan 1, a plurality of feature transformation parameters of the firstnetwork and a plurality of feature transformation parameters of thesecond network being the shared network parameters, and each of thefeature transformation parameters of the first network being anindependent network parameter and each of the feature transformationparameters of the second network being an independent network parameter;or

when the quantity of the feature transformation parameter is greaterthan 1, a plurality of feature transformation parameters of the firstnetwork and a plurality of feature transformation parameters of thesecond network being the shared network parameters, and each of thefeature transformation parameters of the first network being a sharednetwork parameter and each of the feature transformation parameters ofthe second network being a shared network parameter.

In another embodiment, the first network corresponds to a first scale,and the second network corresponds to a second scale; and when the imageprocessing model is called to perform sharpness restoration on theblurry image to obtain a restored image, the one or more firstinstructions are loaded by the processor 301 to perform:

downsampling the blurry image according to the first scale to obtain ablurry image with first sharpness;

activating the first network to perform sharpness restoration on theblurry image with the first sharpness to obtain an intermediate image;

downsampling the blurry image according to the second scale to obtain ablurry image with second sharpness; and

activating the second network to perform sharpness restoration accordingto the blurry image with the second sharpness and the intermediate imageto obtain a restored image.

In another embodiment, the sharpness restoration includes: performingfeature extraction on an image, performing, by using a multi-orderresidual function, feature transformation on an image obtained after thefeature extraction, and performing feature reconstruction on an imageobtained after the feature transformation.

In another embodiment, when network parameters of the first networkand/or network parameters of the second network in the image processingmodel are updated according to the restored image and the clear image,the one or more first instructions are loaded by the processor 301 toperform:

obtaining an optimization function of the image processing model;

determining a value of the optimization function according to therestored image and the clear image; and

updating, according to the principle of reducing the value of theoptimization function, the network parameters of the first networkand/or the network parameters of the second network in the imageprocessing model.

In another embodiment, when a sample pair for training is obtained, theone or more first instructions are loaded by the processor 301 toperform:

obtaining image sequence frames for training, the image sequence framesincluding at least two frames of images;

selecting one frame of image from the image sequence frames as a clearimage, and determining a plurality of frames of reference imagesassociated with the clear image; and

obtaining a blurry image corresponding to the clear image according tothe plurality of frames of reference images, and constructing, by usingthe blurry image and the clear image, the sample pair for training.

In another embodiment, when a blurry image corresponding to the clearimage is obtained according to the plurality of frames of referenceimages, the one or more first instructions are loaded by the processor301 to perform:

superimposing and averaging the plurality of frames of reference imagesto obtain the blurry image corresponding to the clear image.

In another embodiment, one or more second instructions stored in thecomputer storage medium may be loaded and executed by the processor 301to implement corresponding steps of the method in the foregoingembodiments related to image processing. In an implementation, the oneor more second instructions in the computer storage medium are loaded bythe processor 301 to perform the following steps:

obtaining a to-be-processed original image, the sharpness of theoriginal image being less than a preset threshold;

activating an image processing model to perform sharpness restoration onthe original image to obtain a target image, the sharpness of the targetimage being greater than the preset threshold, the image processingmodel at least including a first network and a second network; the firstnetwork corresponding to a first scale, and the second networkcorresponding to a second scale; and the network parameters of the firstnetwork and the network parameters of the second network meeting aselective sharing condition, and the selective sharing condition beingused for indicating shared network parameters between the first networkand the second network, and being used for indicating independentnetwork parameters between the first network and the second network; and

outputting the target image.

In one embodiment, the network parameters include a feature extractionparameter and a feature transformation parameter;

the selective sharing condition, when being used for indicating theshared network parameters between the first network and the secondnetwork, is specifically used for indicating that the featuretransformation parameter of the first network and the featuretransformation parameter of the second network are the shared networkparameters; and

the selective sharing condition, when being used for indicating theindependent network parameters between the first network and the secondnetwork, is specifically used for indicating that the feature extractionparameter of the first network and the feature extraction parameter ofthe second network are the independent network parameters.

In another embodiment, the network parameters further include a featurereconstruction parameter; and

the selective sharing condition, when being used for indicating theindependent network parameters between the first network and the secondnetwork, is further used for indicating that the feature reconstructionparameter of the first network and the feature reconstruction parameterof the second network are the independent network parameters.

In one embodiment consistent with the present disclosure, the imageprocessing model for processing blurry images at least includes thefirst network with the first scale and the second network with thesecond scale. Because there are shared network parameters andindependent network parameters between the first network and the secondnetwork, when performing sharpness restoration on the blurry image, theimage processing model can learn more image features in the blurry imageto obtain a more accurate restored image. The network parameters of thefirst network and/or the network parameters of the second network areupdated according to the more accurate restored image and the clearimage, which may improve the deblurring performance of the trained imageprocessing model. In addition, because there are shared networkparameters between the first network and the second network, thequantity of parameters of the image processing model may be reduced, andthe efficiency of model training is improved. Moreover, by using thecorresponding clear image and blurry image to perform end-to-endtraining and learning on the image processing model, there is no need tosegment the blurry image into blurry regions in movement, and there isno need to make any assumption on the blurry image, which may furtherimprove the deblurring performance of the trained image processing modeland the efficiency of model training.

The term unit, and other similar terms such as subunit, module,submodule, etc., in this disclosure may refer to a software unit, ahardware unit, or a combination thereof. A software unit (e.g., computerprogram) may be developed using a computer programming language. Ahardware unit may be implemented using processing circuitry and/ormemory. Each unit can be implemented using one or more processors (orprocessors and memory). Likewise, a processor (or processors and memory)can be used to implement one or more units. Moreover, each unit can bepart of an overall unit that includes the functionalities of the unit.

What is disclosed above is merely exemplary embodiments of thisapplication, and certainly is not intended to limit the scope of theclaims of this application. Therefore, equivalent variations made inaccordance with the claims of this application shall fall within thescope of this application.

What is claimed is:
 1. A training method for an image processing modelfor processing blurry images, performed by a network device, the imageprocessing model comprising a first network and a second network; thefirst network and the second network being codec networks with differentscales; the sizes of the scales corresponding to the measurements of thesharpness of to-be-processed blurry images; and the method comprising:obtaining a sample pair for training, the sample pair comprising a clearimage and a blurry image corresponding to the clear image; and thesharpness of the clear image being greater than a preset threshold, andthe sharpness of the blurry image being less than the preset threshold;activating the image processing model to perform sharpness restorationon the blurry image to obtain a restored image; and updating networkparameters of the first network and network parameters of the secondnetwork in the image processing model according to the restored imageand the clear image to obtain a trained image processing model; whereinthe network parameters of the first network and the network parametersof the second network meet a selective sharing condition, and theselective sharing condition indicates the network parameters between thefirst network and the second network are shared or independent.
 2. Themethod according to claim 1, wherein the network parameters comprise afeature extraction parameter and a feature transformation parameter; theselective sharing condition, when indicating the network parametersbetween the first network and the second network are shared, indicatethat the feature transformation parameter of the first network and thefeature transformation parameter of the second network are the sharednetwork parameters; and the selective sharing condition, when indicatingthe network parameters between the first network and the second networkare independent, indicate that the feature extraction parameter of thefirst network and the feature extraction parameter of the second networkare the independent network parameters.
 3. The method according to claim2, wherein the network parameters further comprise a featurereconstruction parameter; and the selective sharing condition, whenbeing used for indicating the independent network parameters between thefirst network and the second network, indicates that the featurereconstruction parameter of the first network and the featurereconstruction parameter of the second network are the independentnetwork parameters.
 4. The method according to claim 2, wherein theselective sharing condition indicates that the feature transformationparameter of the first network and the feature transformation parameterof the second network are the shared network parameters comprises: whenthe quantity of the feature transformation parameter is greater than 1,a plurality of feature transformation parameters of the first networkand a plurality of feature transformation parameters of the secondnetwork being the shared network parameters, and each of the featuretransformation parameters of the first network being an independentnetwork parameter and each of the feature transformation parameters ofthe second network being an independent network parameter.
 5. The methodaccording to claim 2, wherein the selective sharing condition indicatesthat the feature transformation parameter of the first network and thefeature transformation parameter of the second network are the sharednetwork parameters comprises: when the quantity of the featuretransformation parameter is greater than 1, a plurality of featuretransformation parameters of the first network and a plurality offeature transformation parameters of the second network being the sharednetwork parameters, and each of the feature transformation parameters ofthe first network being a shared network parameter and each of thefeature transformation parameters of the second network being a sharednetwork parameter.
 6. The method according to claim 1, wherein the firstnetwork corresponds to a first scale, and the second network correspondsto a second scale; and the activating the image processing model toperform sharpness restoration on the blurry image to obtain a restoredimage comprises: downsampling the blurry image according to the firstscale to obtain a blurry image with first sharpness; activating thefirst network to perform sharpness restoration on the blurry image withthe first sharpness to obtain an intermediate image; downsampling theblurry image according to the second scale to obtain a blurry image withsecond sharpness; and activating the second network to perform sharpnessrestoration according to the blurry image with the second sharpness andthe intermediate image to obtain a restored image.
 7. The methodaccording to claim 6, wherein the sharpness restoration comprises:performing feature extraction on an image, performing, by using amulti-order residual function, feature transformation on an imageobtained after the feature extraction, and performing featurereconstruction on an image obtained after the feature transformation. 8.The method according to claim 1, wherein the updating network parametersof the first network and network parameters of the second network in theimage processing model according to the restored image and the clearimage comprises: obtaining an optimization function of the imageprocessing model; determining a value of the optimization functionaccording to the restored image and the clear image; and updating, byreducing the value of the optimization function, the network parametersof the first network and the network parameters of the second network inthe image processing model.
 9. The method according to claim 1, whereinthe obtaining a sample pair for training comprises: obtaining imagesequence frames for training, the image sequence frames comprising atleast two frames of images; randomly selecting one frame of image fromthe image sequence frames as a clear image, and determining a pluralityof frames of reference images associated with the clear image; andobtaining a blurry image corresponding to the clear image according tothe plurality of frames of reference images, and constructing, by usingthe blurry image and the clear image, the sample pair for training. 10.The method according to claim 9, wherein the obtaining a blurry imagecorresponding to the clear image according to the plurality of frames ofreference images comprises: superimposing and averaging the plurality offrames of reference images to obtain the blurry image corresponding tothe clear image.
 11. An image processing method, performed by a networkdevice, the method comprising: obtaining a to-be-processed originalimage, the sharpness of the original image being less than a presetthreshold; activating an image processing model for processing blurryimages to perform sharpness restoration on the original image to obtaina target image, the sharpness of the target image being greater than thepreset threshold, the image processing model at least comprising a firstnetwork and a second network; the first network and the second networkbeing codec networks with different scales; the sizes of the scalescorresponding to the measurements of the sharpness of to-be-processedblurry images; and the network parameters of the first network and thenetwork parameters of the second network meeting a selective sharingcondition, and the selective sharing condition indicating the networkparameters between the first network and the second network are sharedor independent; and outputting the target image.
 12. The methodaccording to claim 11, wherein the network parameters comprise a featureextraction parameter and a feature transformation parameter; theselective sharing condition, when indicating network parameters betweenthe first network and the second network are shared, indicates that thefeature transformation parameter of the first network and the featuretransformation parameter of the second network are the shared networkparameters; and the selective sharing condition, when indicating thenetwork parameters between the first network and the second network areindependent, indicates that the feature extraction parameter of thefirst network and the feature extraction parameter of the second networkare the independent network parameters.
 13. The method according toclaim 12, wherein the network parameters further comprise a featurereconstruction parameter; and the selective sharing condition, whenindicating the network parameters between the first network and thesecond network are independent, indicates that the featurereconstruction parameter of the first network and the featurereconstruction parameter of the second network are the independentnetwork parameters.
 14. A training apparatus for an image processingmodel for processing blurry images, the image processing modelcomprising a first network and a second network; the first network andthe second network being codec networks with different scales; the sizesof the scales corresponding to the measurements of the sharpness ofto-be-processed blurry images; and the apparatus comprising: aprocessor, and a memory connected to the processor, the memory storingmachine-readable instructions, and the machine-readable instructionsbeing executable by the processor to: obtain a sample pair for training,the sample pair comprising a clear image and a blurry imagecorresponding to the clear image; and the sharpness of the clear imagebeing greater than a preset threshold, and the sharpness of the blurryimage being less than the preset threshold; activate the imageprocessing model to perform sharpness restoration on the blurry image toobtain a restored image; and update network parameters of the firstnetwork and network parameters of the second network in the imageprocessing model according to the restored image and the clear image toobtain a trained image processing model; wherein the network parametersof the first network and the network parameters of the second networkmeet a selective sharing condition, and the selective sharing conditionindicating the network parameters between the first network and thesecond network, are shared or independent.
 15. The apparatus accordingto claim 14, wherein the network parameters comprise a featureextraction parameter and a feature transformation parameter; theselective sharing condition, when indicating the network parametersbetween the first network and the second network are shared, indicatesthat the feature transformation parameter of the first network and thefeature transformation parameter of the second network are the sharednetwork parameters; and the selective sharing condition, when indicatingthe network parameters between the first network and the second networkare independent, indicates that the feature extraction parameter of thefirst network and the feature extraction parameter of the second networkare the independent network parameters.
 16. The apparatus according toclaim 15, wherein the network parameters further comprise a featurereconstruction parameter; and the selective sharing condition, whenindicating the network parameters between the first network and thesecond network are independent, indicates that the featurereconstruction parameter of the first network and the featurereconstruction parameter of the second network are the independentnetwork parameters.
 17. The apparatus according to claim 15, wherein theselective sharing condition being specifically used for indicating thatthe feature transformation parameter of the first network and thefeature transformation parameter of the second network are the sharednetwork parameters comprises: when that the quantity of the featuretransformation parameter is greater than 1, a plurality of featuretransformation parameters of the first network and a plurality offeature transformation parameters of the second network being the sharednetwork parameters, and each of the feature transformation parameters ofthe first network being an independent network parameter and each of thefeature transformation parameters of the second network being anindependent network parameter.
 18. The apparatus according to claim 15,wherein the selective sharing condition being specifically used forindicating that the feature transformation parameter of the firstnetwork and the feature transformation parameter of the second networkare the shared network parameters comprises: when the quantity of thefeature transformation parameter is greater than 1, a plurality offeature transformation parameters of the first network and a pluralityof feature transformation parameters of the second network being theshared network parameters, and each of the feature transformationparameters of the first network being a shared network parameter andeach of the feature transformation parameters of the second networkbeing a shared network parameter.
 19. The apparatus according to claim14, wherein the first network corresponds to a first scale, and thesecond network corresponds to a second scale; and the activating theimage processing model to perform sharpness restoration on the blurryimage to obtain a restored image comprises: down sampling the blurryimage according to the first scale to obtain a blurry image with firstsharpness; activating the first network to perform sharpness restorationon the blurry image with the first sharpness to obtain an intermediateimage; down sampling the blurry image according to the second scale toobtain a blurry image with second sharpness; and activating the secondnetwork to perform sharpness restoration according to the blurry imagewith the second sharpness and the intermediate image to obtain arestored image.
 20. The apparatus according to claim 19, wherein thesharpness restoration comprises: performing feature extraction on animage, performing, by using a multi-order residual function, featuretransformation on an image obtained after the feature extraction, andperforming feature reconstruction on an image obtained after the featuretransformation.